How to do web scraping with Flutter

How to do web scraping with Flutter

In the realm of digital innovation, web scraping or web crawler has emerged as a powerful tool, revolutionising the way we access and analyse data. In this article, we are exploring its application in the context of the National Football League (NFL) and shedding light on a pioneering Flutter app that harnesses this technology to provide users with a unique and immersive NFL experience.

The Essence of Web Scraping:

Its essence, is the automated extraction of data from websites. It enables the retrieval of valuable information, transforming raw data into structured and meaningful insights. In the context of sports, web scraping allows enthusiasts to access real-time scores, player statistics, and other pertinent details without the need for manual data entry.

Bridging the Gap in Flutter: 

By leveraging this technology, This app scrapes data from various NFL sources, aggregating a comprehensive repository of information. This includes only super bowl history for now just to show you an example, all presented in a native and user-friendly interface.

The Advantages of Web Scraping in Flutter:

1. Real-Time Updates: It enables Flutter app to provide users with live, up-to-the-minute updates engaging experience without web APIs.
2. User-Friendly Interface: The Flutter framework creates a seamless and intuitive user interface.

Example of Scraping

Use web_scrapper plugin to scrap the data from website into flutter.

import 'package:flutter/material.dart';
import 'package:url_launcher/url_launcher.dart';
import 'package:web_scraper/web_scraper.dart';

void main() => runApp(WebScraperApp());

class WebScraperApp extends StatefulWidget {
  @override
  _WebScraperAppState createState() => _WebScraperAppState();
}

class _WebScraperAppState extends State<WebScraperApp> {
  // initialize WebScraper by passing base url of website
  final webScraper = WebScraper('https://webscraper.io');

  // Response of getElement is always List<Map<String, dynamic>>
  List<Map<String, dynamic>>? productNames;
  late List<Map<String, dynamic>> productDescriptions;

  void fetchProducts() async {
    // Loads web page and downloads into local state of library
    if (await webScraper
        .loadWebPage('/test-sites/e-commerce/allinone/computers/laptops')) {
      setState(() {
        // getElement takes the address of html tag/element and attributes you want to scrap from website
        // it will return the attributes in the same order passed
        productNames = webScraper.getElement(
            'div.thumbnail > div.caption > h4 > a.title', ['href', 'title']);
        productDescriptions = webScraper.getElement(
            'div.thumbnail > div.caption > p.description', ['class']);
      });
    }
  }

  @override
  void initState() {
    super.initState();
    // Requesting to fetch before UI drawing starts
    fetchProducts();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Fetch Data Example',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: Scaffold(
          appBar: AppBar(
            title: Text('Product Catalog'),
          ),
          body: SafeArea(
              child: productNames == null
                  ? Center(
                      child:
                          CircularProgressIndicator(), // Loads Circular Loading Animation
                    )
                  : ListView.builder(
                      itemCount: productNames!.length,
                      itemBuilder: (BuildContext context, int index) {
                        // Attributes are in the form of List<Map<String, dynamic>>.
                        Map<String, dynamic> attributes =
                            productNames![index]['attributes'];
                        return ExpansionTile(
                          title: Text(attributes['title']),
                          children: <Widget>[
                            Padding(
                              padding: const EdgeInsets.all(10.0),
                              child: Column(
                                children: <Widget>[
                                  Container(
                                    child: Text(
                                        productDescriptions[index]['title']),
                                    margin: EdgeInsets.only(bottom: 10.0),
                                  ),
                                  InkWell(
                                    onTap: () {
                                      // uses UI Launcher to launch in web browser & minor tweaks to generate url
                                      launch(webScraper.baseUrl! +
                                          attributes['href']);
                                    },
                                    child: Text(
                                      'View Product',
                                      style: TextStyle(color: Colors.blue),
                                    ),
                                  ),
                                ],
                              ),
                            )
                          ],
                        );
                      }))),
    );
  }
}

Conclusion: As we navigate the intersection of technology and sports, web scraping emerges as a game-changer, providing unprecedented access to real-time data. 

Find more about my expertise here.