trying to extract two tables at a time from http files with 40 tables each

Question

I have about 20 webpages. Each page has top banner navigation and then has information on up to 20 vehicles. There are 2 tables per vehicle.
The logical flow is: page navigation, table 1 for vehicle 1, table 2 for vehicle 1, table 1 for vehicle 2, table 2, for vehicle 2, ... end of page.
Example of tables included below.

I want to get the information out of the html pages and into a database.
The plan: separate out the data for each individual vehicle into individual files & then parse/extract the data from the files.

I do not understand awk so I am using sed.

Extraction plan: find the line w/ "car_photo", go back 4 lines (which will be the table tag), extract from that line until the second /table tag. Repeat until the final set of tables.

I've looked online for examples of how to get sed to extract from a given line number until the next instance of a regular expression... it keeps extracting until the final instance. & even if it did work, I want it to extract up to the 2nd instance.

Here is a sample of a file, with the data replaced w/more generic info.

---

32321 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:
32322 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:
32321 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:
32323 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:
32324 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:
32325 Make: Model:
Year:
VIN:
Color:
Year Acquired:
Mileage: Last Oil Change: Insurance Due: Registration Expires:

---

I tried to create a loop that would run 20 times. Each time, sed would extract lines 1 through the line with </table> then sed runs again to delete those lines. It then extracts lines 1 through the next line with </table> again (to get the 2nd table) sed then deletes the second table.

Each time `sed` extracts a table, it concatenates to a new file using the loop counter.

The problem is that sed is not stopping at the first occurrence of </table>. It is stopping at the LAST occurrence.

Answer 1

If I were doing this often, I would be using XPath parsing via something like the Nokogiri gem for Ruby.

However, here's something that could work, but without a bash script to combine them will require a couple of steps per file ( I guess that's 20 in your case ).

Step 1: Convert the html into line by line as much as possible so that awk can process it.

Starting with your comment's html input into car.html, I did

cat car.html | awk -F"> " '{ for( i = 1; i <= NF; i++ ) printf( "%s>\n", $i ) } ' > new.html

which gave me a new.html file like

<table>
<tr><td width="90">
<div class="car_photo">
<div class="space">
<img src="../photos/veh5.jpeg">
</div>
</div>
</div>
</td>
<td align="right" class="car_details" width="400">
<table>
<tr>
<td class="line_bottom" width="190">
<div class="text_left">32325</a>
</div>
</td>
</tr>
</table>
<div class="line_bottom">
Make: </div>
<div class="line_bottom">Model: <br>Year: <br />
</div>
</td>
<td class="car_details" width="400">
<div class="line_bottom">Mileage:</div>
<div class="line_bottom">Oil Change: </div>
<div class="line_bottom">Registration:</div>
<br>
</td>
</tr>
</table>

Step 2: Take that file and put it through an awk script I put into it's own file called awko

#!/usr/bin/awk -f

BEGIN { FS=">" }

$1 ~ /<table/ { table_cnt++ }

$1 ~ /<\/table/ { table_cnt-- }

table_cnt > 0 {
    for( i = 1; i <= NF; i++ ) {
        split( $i, arr, "<" )
        if( length( arr[ i ] ) > 0 )
            printf( "%s\n", arr[ 1 ] )
    }

}

running this like

awko new.html 

gave me a result like:

32325
Make: 
Model: 
Mileage:
Oil Change: 
Registration:

The output in the awko could be modified to make a CSV styled output instead to make it easier to import into a DB. And again, these different steps could be combined in a shell script to to the "heavy filename lifting" in a proper loop, but I don't have time for that now.

awko is essentially searching for what is the text of each line with the start/end you specified.

Oops. I just noticed this question is old. Oh well, committing this answer anyway.