Ok, I’m overdue for a ‘weekly’ status report. I’m going to try to kick this back into gear as it helps you out there track things and me feel like I’m moving forward by writing down all the little things I’ve done over the last bit.
Trello board to keep up to date: https://trello.com/board/bookie/4f18c1ac96c79ec27105f228
In an effort to add some features to Bookie I’ve ended up starting two new repos of code meant to interact with Bookie.
This is meant to start taking over the work of reading the page content and readable parsing the important content out. It was a chance to play with Tornado and Heroku. This also means that in the future I’ll be able to scale out the readable processing serperatly from the main Bookie website and host. It’s pretty bare bones right now and doesn’t directly talk to Bookie, but I’ll look at adding that integration soon as the API stabilizes and I get more tests going in it.
So far the Heroku bit has been pretty awesome. I have to deal with the fact that the app gets shut down and has to restart on first request, but hopefully that gets better as traffic and use picks up. You can tinker with it at http://readable.bmark.us
I’ve been wanting to start up a command line client for some of the Bookie work. The big thing is that I need tools to help manage invites and such. So it’s currently very admin centric, but eventually I’d like to get this into a ncurses cool command line interface to pull up recent bookmarks and even do some quick searches via the API. Aren’t API’s cool. This will also contain the reference Python API implementations so we’ll have two implementations soon. One in JS and one in Python.
I’ve got a beta version (which is really an alpha) up on PyPi so you can
$ pip install bookie_api $ bookie ping
Build baby build
I spent some quality time with http://build.bmark.us to get the JS tests running via grover and phantomjs and that’s awesome. I also added the new projects into the builder as well. So, while I don’t have all the tests I need, at least now the ones I do have run consistantly.
Other little tweaks
- Prettied up the new user invite email and landing page
- Fixed a bug with dupe tags in the tagcontroller
- Added more icons from the fontawesome set to pretty up the ui, especially the account page.
- Lots of changes to the make/build steps for JS and CSS including actually doing the pyscss transition.
- Everything is now on the final stable release of YUI 3.5. It’s been a good ride through the development releases.
I’ll be giving a talk at Penguicon on using YUI for JS app development. If you’re in the area stop by. This is Friday April 27th, at 6pm. Then on Saturday I’ve got a Bookie mini-sprint going on. I’ll probably be hacking most of the weekend. Feel free to stop by and check things out.
Well, with all the great stuff going on with Bookie, I’ve gotten a bit buried in some big changes. The background processing and importing updates are going to take a bit to get right.
This means, there’s a great chance for others to hack up the little tweaks that we need to really add some polish to Bookie. So below I’ve listed a few ideas that should be pretty simple things to add, but with a really good positive and visible effect on the site.
Add notification that user has invites
Now that invites are there, we should highlight a user’s account navigation link to let them know they have invites available. I’ll periodically add them to the system, and we don’t want users to have to go to their account page each time to see they’ve got invites. I think a simple adding of one of the envelope/message type icons from our font-icon set would be perfect, with some sort of hover message to start. We might also want to highlight the block in their account page so it stands out that the invites are available.
Flash message system.
We want to be able to let users know things have happened successfully after doing something that redirected them. Imports are going to be doing this, saving/updating bookmarks, etc. It’d be nice to have a consistent type of ui to drop flash messages in and them to show after a redirect.
Show new user message if self bookmarks page has no results
When a new users starts up and logs in, they default to their own page of bookmarks…which is going to be empty. So we should detect this in our JS code that fetches the results and displays a set of default content with links to things like importing instructions, where to get the chrome extensions, and other handy new user tips.
Some of this might also be nice to use for the email that a new user gets when they’ve been invited to Bookie.
Add firefox bookmark importer
Ok, so this one is a bit more involved, but really, it’s a single class and a couple of Python methods. The hard part is reading in and figuring out how to match bookmarks to tags in Firefox’s JSON dump of bookmarks. Once we get the Firefox extension rolling, it’ll be great to have a good import system for the browser as well.
Well, here are four things I’d love to see happen in the near future to help make the experience a level nicer for everyone. If you’re interested in all or have any questions, ping me in #bookie in irc or shoot me a comment below. I’d be happy to help walk anyone through these or any other ideas you might have.
Phew, tiring trip to PyCon this year. This was my second year after hitting up my first last year. The conference definitely felt larger than last year as they crossed 2,200 attendees. It’s unbelievable to see how large the Python community has gotten. I can’t stress what great job the people that put this together.
Last year I hardly knew anyone. This year, however, I got to put faces to people I’ve interacted with over the last year, welcome back those I met last year, and get some face to face time with new co-workers from Canonical. The social aspect was a larger chunk of my time this year for sure.
Side note, I listen to The Changelog podcast from time to time, and I love their question on who you’d love to pair up/hack with as a programming hero type question. I got to meet and greet mine at this PyCon by meeting up with Mike Bayer. He’s behind some great tools like SqlAlchemy and Mako. What I love is that, not only does he rock the code part, but the community part as well. I’m always amazed to see the time he puts into his responses to questions and support avenues. Highlight of my PyCon for sure.
I’ll post a seperate blog post on my sprint notes. I feel that if you’re going to go, you might as well stay for sprints. I get as much out of that as the conference parts itself. I think I made some good progress on things for Bookie this year. The big thing is that an invite system is in place, so if you’d like an account on Bmark.us let me know and I’ll toss an invite your way.
- Introduction to Metaclasses
- Basic but reminded me how the bits worked and had some good examples. I like this because I often write ‘the code I want to be writing’ and then write my modules/etc to fit and metaclasses help with this sometimes.
- Fast Test, Slow Test
- Just a reminder that fast tests are true unit tests and run during dev which helps make things easier/faster as you go vs the whole ‘mad code’ then wait for feedback on how wrong you are.
- Practical Machine Learning in Python
- mloss.org – check out for lots of notes/etc on ML in OSS
- ml-class.org – teach me some ML please
- sluggerml – app he built as a ML demo
- scikit-learn : lots of potential, very active right now
- Introduction to PDB
- whoa…where have you been all my life ‘until’ command?
- use ‘where’ more to move up stack vs adding more debug lines
- Flexing SQLAlchemy’s Relational Power
- Hand Coded Applications with SQLAlchemy
- <3 SqlAchemy. Some really good examples of writing less code by automating the biolerplate with conventions.
- Web Server Bottlenecks And Performance Tuning
- lesson: if you think it’s apache’s fault think again. You’re probably doing it wrong.
- Advanced Celery
- check out cyme https://github.com/celery/cyme, possible way to more easily run/distribute celery work?
- cool to see implementations of map/reduce using celery
- chords and groups are good, check them out more
- Building A Python-Based Search Engine
- Good talk for into into terms and such for fulltext search
- Lighting talks of note
- py3 porting docs: http://docs.python.org/howto/pyporting
- bpython rewind feature is full of win over ipython
- ‘new virtualenv’ trying to get into stdlib for py3.3, cool!
- asyncdynamo cool example of async boto requests for high performance working with AWS api (uses tornado)
- I WANT the concurrent.features library…but it’s Python 3 :(
I’m very late on a bunch of weekly status reports, but I’ve got good reason I promise. This summer has been a bit of a “summer of change” for me. Perhaps more renovation than change.
First, I decided I was sick of looking and feeling like crap and got into a weight loss program. I’ve finished that program today with some really good results. I’m down 37lbs, with another 19 to go until I hit my goal. Here’s hoping I can get there before the year is out. This has really helped me feel a ton better. I did a lot of biking this summer, including a 48mile monster that beat me up, but man was that fun.
Next up, I finally managed to clean the garage out. That means the woodworking tools are open for business. I’ve only started working on putting some finish on some new oak closet doors, but that’s a huge move forward. I’ve not done any woodworking since the boy was born and now that the space is cleared out I’m looking forward to getting back to making some shavings.
I can’t express how excited I am to get this chance to work with a team of smart people that are really working hard to build something awesome. In many ways this is the culmination of a goal I set years ago to get paid to work on open source software, and not only that, but to get paid to work with Python as well.
Morpace was really good to me and I feel sad to leave them. They gave me my first chance to prove that I could be a productive Python developer. I’ve grown though and the opportunity to work with all of the great people at Canonical is just too much to pass up. I can’t wait to jump in and see what I can help with.
It’s been a good summer. I can only hope that each year continues to be as fulfilling.
Phew, it’s been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There’s now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don’t forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.
We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we’re running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you’re logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.
We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can’t get a cool url for Bookie such as http://bookie.net.
With the new site I purchased an SSL certificate. So you’ll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync’d off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.
All of the code updates are available in the develop branch on Github. If you’ve got any questions feel free to hop into #bookie on irc and let me know.
With all this going on I’m looking for some early alpha testers. If you’re interested in Bookie, but couldn’t, or didn’t want to, setup your own instance, let me know. I’d love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.
Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.
If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie
- Current Chrome Extension: 0.3.3
- Most recent code updates: develop branch
- Current working branch: feature/auth
I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you’ve accomplished and what you plan into the public that helps keep the motivation motor running.
In an effort to keep Bookie from stagnating, I think that’s a good thing to start doing. Count this as the first of a series of weekly progress reports I’m going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.
This past week has been a bit crazy. There hasn’t been a ton of time to put into things, but I’ve managed to move a few big things forward:
First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.
In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There’s a ton of work in getting this going, but it’s a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that’s really the big goal that I’ve set myself by the end of the year, I’m feeling good on this one.
The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It’ll never scale that way. So I’ve split the work into a couple of parts.
One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I’m able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.
I’m definitely looking forward to ramping this up on a real server with Postgresql running. I’d love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.
So that’s this week’s report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!
If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie
Introducing Bookie: only for developers
Delicious has been pretty stagnant over the years. I love using Instapaper and really kept waiting for Delicious to add that feature in to my pages I send into Delicious, but it never arrived. I also kept waiting for link checkers, a decent mobile view, and maybe even some support for full text search of actual web page content. Now that Yahoo has leaked out that Delicious is going to be “Sunset”, I’ve finally gotten the push to start up my own bookmark storing application that I can add all the great features I want to have in there.
This is the 0.1 release I’m calling “only for developers”. It’s not an easy one-click install, and right now it’ll want you to have the -dev packages for the db engines to work right, but it does work. It can store, retrieve, and search bookmarks stored. It supports Delicious and Google bookmarks import, and will do an export. There’s a fully functioning Google Chrome extension you can use to store and edit bookmarks.
This is currently a self hosting application. In time, I hope to provide a hosted version perhaps. I also hope to get a Firefox extension, as well as get started on the fancy features I noted above. So this is a LONG way from complete, but I feel like I’ve hit at least a 0.1 worthy milestone and want to put a stake in the ground. Here’s the start of something, and hopefully it’ll keep improving in future releases.
I want to thank my local guys for helping, testing, and adding great bits of code to bookie already. It’s a lot of fun seeing other people get interested in things and contributing as well.
I’m trying to keep the todo list and some idea of the next milestones in the github wiki. Feel free to note bugs, add requests, etc. You can also chat with us in #bookie on freenode.