Start a new topic
Answered

How to pull data from external site to display?

Hello,


I want to set up a LaMetric app to pull data (for example, the current view count from a blog page).  How do I go about creating this?  Does LaMetric only access information pushed specifically to it, or can it "scrape" fro a site's HTML?


I would expect this to be a "poll" request, given as how the LaMetric box would need to request updates from the server, but the interface doesn't make it clear how to select which data will be pulled - and the "validate" button doesn't appear to have any functionality.


Any help is greatly appreciated!


Best Answer

@Westwise, yes. It runs curl requests to get the data then returns it in a format LaMetric understands.


@Kristof Toth here you go; I genericized it some, but it should help you get the idea 

<?php
// Frame index
$idx = 0;

// you'll want icons equal to your number of frames here
$icon = array();
$icon[0] = 'i605';
$icon[1] = 'i606';
// this is for connection errors
$iconerr = 'i1059';

// Work to pull information from first source goes here
// (I use curl, and $response is the data via CURLOPT_RETURNTRANSFER)
// ...
// ...

// Assuming the server responded, process it
if ($response) {
  $data = json_decode($response,true);
  // Build frames array
  $out = array('frames'=>array(
               array('index'=>$idx,
                     'text'=>$data['value'],
                     'icon'=>$icon[$idx])
  ));
// Something went wrong with the connection!
} else {
  // Build frames array with error message
  $out = array('frames'=>array(
               array('index'=>$idx,
                     'text'=>'Error',
                     'icon'=>$iconerr)
  ));
}

// on to the next frame...
++$idx;

// Work to pull information from second source goes here
// (same basic process as above)
// ...
// ...

if ($response) {
  $data = json_decode($response,true);
  // I only want this frame to show if it has a nonzero value...
  if ($data['value'] > 0) {
    // Add onto the frames array
    array_push($out['frames'], array(
      'index'=>$idx,
      'text'=>$data['value'],
      'icon'=>$icon[$idx])
    ));
  // If it's zero, duplicate the previous frame
  // (LaMetric does not currently support dynamic numbers of frames)
  } else {
    array_push($out['frames'],$out['frames'][($idx - 1)]);
    $out['frames'][$idx]['index'] = $idx;
  }
}

// LaMetric plays best with pretty-printed JSON...
$outjson = json_encode($out, JSON_PRETTY_PRINT);
echo $outjson;
?>

 


1 person has this question

You need a JSON formatted response from your webserver that follows the format specified. 


For example if your url responds with:  

{
    "frames": [
        {
            "index": 0,
            "text": "950000",
            "icon": "i44"
        }
    ]
}

 Then 950000 is displayed in frame 1(0) with the Kickstarter icon.


I've left a test file for people at: http://ahref.co.uk/clock/test.json


Add it to an app make it private, publish it, install it and you'll see the message from the json file.


If your url validates as json(try jsonlint.com) then it'll work.

What about parsing JSON api feeds? How is that best done?

I don't believe apps can currently "scrape" existing content; right now they can only process properly-formatted JSON with the content that they're expecting (as shown in the "Data Format" sample box when you're developing the app).


What I've done to process output of other APIs is create a "shim" application; right now it's a polled application, which fetches the data from the desired API whenever the LaMetric app queries it. Once it has the data from the desired API, it converts it to the JSON format that LaMetric expects and hands that back.

@Jesse Jones so do you host your "shim" application on your own server, to send the polled information to the LaMetric app?

@Jesse Jones, Mind sharing?

Answer

@Westwise, yes. It runs curl requests to get the data then returns it in a format LaMetric understands.


@Kristof Toth here you go; I genericized it some, but it should help you get the idea 

<?php
// Frame index
$idx = 0;

// you'll want icons equal to your number of frames here
$icon = array();
$icon[0] = 'i605';
$icon[1] = 'i606';
// this is for connection errors
$iconerr = 'i1059';

// Work to pull information from first source goes here
// (I use curl, and $response is the data via CURLOPT_RETURNTRANSFER)
// ...
// ...

// Assuming the server responded, process it
if ($response) {
  $data = json_decode($response,true);
  // Build frames array
  $out = array('frames'=>array(
               array('index'=>$idx,
                     'text'=>$data['value'],
                     'icon'=>$icon[$idx])
  ));
// Something went wrong with the connection!
} else {
  // Build frames array with error message
  $out = array('frames'=>array(
               array('index'=>$idx,
                     'text'=>'Error',
                     'icon'=>$iconerr)
  ));
}

// on to the next frame...
++$idx;

// Work to pull information from second source goes here
// (same basic process as above)
// ...
// ...

if ($response) {
  $data = json_decode($response,true);
  // I only want this frame to show if it has a nonzero value...
  if ($data['value'] > 0) {
    // Add onto the frames array
    array_push($out['frames'], array(
      'index'=>$idx,
      'text'=>$data['value'],
      'icon'=>$icon[$idx])
    ));
  // If it's zero, duplicate the previous frame
  // (LaMetric does not currently support dynamic numbers of frames)
  } else {
    array_push($out['frames'],$out['frames'][($idx - 1)]);
    $out['frames'][$idx]['index'] = $idx;
  }
}

// LaMetric plays best with pretty-printed JSON...
$outjson = json_encode($out, JSON_PRETTY_PRINT);
echo $outjson;
?>

 


2 people like this
Hi, I have an app which was originally a push app, but I added pull functionality as well. The URL that is activated when the action button is pressed, returns JSON with frames in the same format as the push data. But when I press the action button the LaMetric only displays a check Mark - no frames are shown with the data from en endpoint. Any ideas on what I'm doing wrong? -Louise
Ok it works if I: -In the URL, which is activated when the action button is pressed, I push data to the LaMetric. I just thought it was enough to return the frames from the pull URL, but as mentioned above, this does not work. Do you agree, or did I misunderstand something? -Louise

Guys, response from request did with action button is ignored. But good news is that now it triggers poll request each time you press the button. We have a tutorial that demonstrates how to create poll app with OAuth2 authorization here. May be helpful.

The Online-Scraper kimonolabs.com allows one to build Webscrapers, which provide their own api in turn. You can even manipulate the output of the kimonolabs-apis to have them output lametric-ready json-blobs. See Modify Results here:

https://help.kimonolabs.com/hc/en-us/articles/204088270-Examples-Modify-Results-Function


So in Theorie it should be possible to build a webscraper with kimono, which outputs lametric-frames. 

To pull data from an external site and display it, you typically use APIs (Application Programming Interfaces) or web scraping techniques, depending on the site's structure and data availability. If the site offers an API, you can make HTTP requests (e.g., GET requests) to retrieve data in formats like JSON or XML, which can then be parsed and displayed on your website or application. Web scraping involves extracting data directly from the HTML of a webpage https://rubmd.net/  using tools like BeautifulSoup or Puppeteer. It’s important to ensure compliance with the site's terms of service when using these methods.

Login or Signup to post a comment