Comments on: scrAPI cheat sheet

By: Arvind

Arvind — Sat, 03 May 2008 06:09:37 +0000

You have done an incredible job with scrAPI. I am learning Ruby mainly to take advantage of your library. So far, I have only scraped one complex page and the process was smooth enough so far. I have to see if my code is going to be maintainable going forward.

One thing I tried is enhancements suggested at http://www.quarkruby.com/2008/1/30/scrapi-enhancements/. I liked the idea but ran in to couple of issues with those changes. Are you planning to implement such feature? Their blog has also helped in using scrAPI.

One other thing I was wondering was can you point me to performance tips in using scrAPI and Tidy so that I can make sure my code is perf conscious. I hear horror stories about ruby performance in general.

Also, I noticed there is not much activity on scrAPI. Is there a plan to improve/maintain going forward.

Thank you for all your efforts in building scrAPI.

By: Assaf

Assaf — Sat, 06 Jan 2007 21:57:40 +0000

Scraper.define do
array :tags
result :tags
process “a[href][rel~=tag]“, :tags=>”@href”
end

By: Edgar

Edgar — Sat, 06 Jan 2007 18:57:20 +0000

I’m just downloading scrAPI

What is the best way to retrieve all the tags (rel=tag) in every post in a feed?

By: Assaf

Assaf — Mon, 11 Sep 2006 16:05:20 +0000

In CSS that would be: E>F+D

By: gik

gik — Mon, 11 Sep 2006 15:55:47 +0000

Assaf,
I know about sources of CSS specification:)
But I propouse to add this selector to scrApi, not in CSS :)

By: Assaf

Assaf — Mon, 11 Sep 2006 15:41:21 +0000

gik, CSS selectors are not part of the Ruby language, but part of the CSS specification. You can learn more about them in this article: http://www.xml.com/pub/a/2003/06/18/css3-selectors.html

By: gik

gik — Mon, 11 Sep 2006 10:16:31 +0000

…
E { F D – an D element which is child of E element parent of F element.

This will allow to scrape content from markup like this:

[e]
[f][/f]
[d]needed content[/d]
[/e]

[e]
[z][/z]
[d]fake content[/d]
[/e]

[e]
[x][/x]
[d]fake content[/d]
[/e]

By: gik

gik — Mon, 11 Sep 2006 10:15:10 +0000

Again….
E

By: gik

gik — Mon, 11 Sep 2006 10:13:48 +0000

Hello Assaf,
I am very new in ruby, and don’t know language so well:)
As I could see it will be very usable to have selector like this:
“E

needed content

fake content

By: Assaf

Assaf — Fri, 08 Sep 2006 21:58:39 +0000

Chris,

Not from the scrape method itself, but you can use Net::HTTP to make a POST request and pass the result body to the scrape method.

If enough people think it’s useful, I can add convenience methods that would do POST and handle form submissions.