Posted by Qamir on 18 November 2010.
November’s meetup saw a great turn out to see two excellent talks by Declan McGrath & Ana Nelson covering scraper wiki & dexy respectively with a some deep conversations after over a beer or two. Here’s my round up of the evening:
Scraper wiki is a tool that allows anyone to create scrapers with all the coding done in the browser, from the scraperwiki website here’s a quick about scraperwiki:
Online code editor, write debug and run Python, PHP or Ruby in your browser
ScraperWiki code libraries make scraping, parsing and geocoding simple
Store data direct to the datastore
API access to data in XML, JSON, PHP, YAML or CSV
Schedule scrapers to re-run so your data is always up-to-date
Email alerts if your scrapers fail
Built in source control
Watch out for a more in depth post by Declan shortly as he delves more into scraperwiki.
Ana presented a talk in dexy a tool she’s developed to make documentation of software projects a lot easier. Ana’s developed dexy as open source project. The potential uses for dexy in automating the documentation process are endless. So make sure you check it out here, there’s also twitter channel there.
Tech mentions during the talks included :
Watir - A simple and flexible open-source tool for automating web browsers.
XLRD - An excel library written in python
Ana mentioned propublica as an great of a site collaboration between developers and SME’s (Subject Matter Experts) in this case journalists which further echoed Declan’s point during his talk whereby subject matter experts who need better & more contextualised information need the help of developers to make it happen and tools such as scraperwiki help hugely to organise and collate information.