Building a web app around some existing code
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I need to be able to download my flight details from tripit into CSV or XLS. Other sites that do it don't give me enough data, or closed down.

A bunch of developers have done great work getting the code ready that can do it; but it's too hard for me to use, and other people might want something easier.

So far, there has been a fair bit of work on this at a code level and it's working really well, but I need it converted into a web app (I'd live with a desktop app) that's easy to use. One dev got close (pretty much there but for some auth bugs) but had to go study. Because I only use it every six months, I keep forgetting how and then it takes days - thus need an app. I'm not a developer.

Ideally, the developer would also use some code I've seen (but don't know where to find) that allows you to select field order and filter. I think datatables.net. The filter would be a date filter (there are already plenty of switches in the code)

The code that does it with Github and Python (and some requirements too) is here..

Bountify Link: https://bountify.co/tripit-api-aimed-at-cytebode
Github: https://github.com/ahmedengu/Tripit_Flights_Export

A sample CSV file from TripIt would be really helpful in order to test the code.
kostasx 3 months ago
Here you can find two files. One is the raw extract I get from the code. The other is what I use. That's what datatables could be very helpful. But the code is already brilliant to be honest. It gives lots of options of what to pull - and if it's there I reckon why not use it... https://www.dropbox.com/sh/h7k2q63lv702s75/AADm4Vw47anQb5qe-sxiE8Ioa?dl=0
sebmack 3 months ago
@sebmack By the way, I found a bug in the code I previously gave you while reviewing it. In activities_to_csv.py, line 112, after except StopIteration:, it should be a continue instead of a break. The bug would make the script to fail if there are no future trips. Maybe that's why you've been having a hard time using the script? It escaped me before because I had a future trip while testing, but now that all my trips are past trips, I see that the script was broken.
CyteBode 3 months ago
Thanks Cytebode! @ahmedengu you may want to retry your fix after seeing Cytebode's code fix?
sebmack 3 months ago
@cytebode. Sorry for the delay. Away from home so cant try it til the weekend. Few questions... 1. Does it use the switches and date filters from your app... I like those? 2. How hard to pass the output to datatables.net to allow filtering/sorting? 3. If i wanted to make it live, what would i need?
sebmack 3 months ago
@sebmack I guess you're not looking at your emails while you're away. I actually answered your questions below 2 days ago, in the comments of my solution, where discussion concerning it rightfully belongs. It's really annoying to be notified of a conversion one's not a part of (the @username doesn't do anything), so I didn't reply to the bounty itself to avoid bothering kostasx. I updated the solution to integrate DataTables, so please check it out.
CyteBode 3 months ago
awarded to CyteBode

Crowdsource coding tasks.

1 Solution


This is a small server that is meant to be run locally (see instructions at the bottom). The door is left open for a deployment on an actual server.

Requirements

  • Python 2.7
  • tornado
  • lxml
  • requests
  • requests_oauthlib
  • unicodecsv

All the required libraries can be installed with pip.

Files

Put the following files in the same directory:

server.py

import os
from StringIO import StringIO
from urllib import urlencode

import tornado.ioloop
import tornado.template
import tornado.web

import lxml.etree as etree
import requests
from requests_oauthlib import OAuth1, OAuth1Session
import unicodecsv as csv

from extraction import EXTRACTORS


API_TOKEN  = "PUT YOUR API KEY HERE"
API_SECRET = "PUT YOUR API SECRET HERE"
APP_NAME   = "put your app name here"


PERIOD_TO_PASTS = {
    "past": ("true", ),
    "future": ("false", ),
    "both": ("false", "true")
}

PAST_TO_PERIOD = {
    "true": "past",
    "false": "future"
}

TRAVELERS = {
    "true": ("true", ),
    "false": ("false", ),
    "all": ("true", "false")
}

TRIP_ID_XPATH = "//Trip/id/text()"
MAX_PAGE_XPATH = "//max_page/text()"


def get_trips_from_api(auth, traveler = "true", period = "both", from_page = 1):
    for trvl in TRAVELERS[traveler]:
        for past in PERIOD_TO_PASTS[period]:
            page_num = max(from_page, 1)
            while True:
                url = "".join(["https://api.tripit.com/v1/",
                    "list/trip/",
                    "traveler/%s/" % trvl,
                    "past/%s/" % past,
                    "format/xml/",
                    "page_num/%d/" % page_num,
                    "page_size/1/",
                    "include_objects/true"
                ])

                response = requests.get(url, auth = auth)
                tree = etree.fromstring(response.text)

                try:
                    max_page = int(tree.xpath(MAX_PAGE_XPATH)[0])
                    trip_id = tree.xpath(TRIP_ID_XPATH)[0]
                    yield tree

                    if page_num >= max_page:
                        break
                except IndexError:
                    break

                page_num += 1


oauth_sessions = {}


class LoginHandler(tornado.web.RequestHandler):
    def get(self):
        if (self.get_secure_cookie("session_id") in oauth_sessions):
            self.redirect("/main")
        else:
            self.write(open("login.html").read())


class TripitHandler(tornado.web.RequestHandler):
    def get(self):
        referer = self.request.headers.get('Referer')
        try:
            REQUEST_TOKEN_URL = "https://api.tripit.com/oauth/request_token"
            oauth = OAuth1Session(API_TOKEN, client_secret=API_SECRET)
            token = oauth.fetch_request_token(REQUEST_TOKEN_URL)

            BASE_AUTHORIZATION_URL = "https://www.tripit.com/oauth/authorize"
            authorization_url = oauth.authorization_url(BASE_AUTHORIZATION_URL)

            oauth_sessions[token["oauth_token"]] = {
                "session": oauth,
                "oauth_token": token["oauth_token"],
                "oauth_token_secret": token["oauth_token_secret"]
            }
            self.set_secure_cookie("session_id", token["oauth_token"])

            self.redirect(authorization_url + "&" + urlencode({
                "oauth_callback": referer + "oauth"
            }))
        except Exception as e:
            print(traceback.format_exc())
            self.clear_cookie("session_id")
            self.redirect("/")


class OAuthHandler(tornado.web.RequestHandler):
    def get(self):
        req = self.request

        oauth_token = self.get_argument("oauth_token")
        session = oauth_sessions[oauth_token]

        url = "%s://%s%s?oauth_token=%s" % (
            req.protocol, req.host, req.uri, oauth_token
        )
        oauth = session["session"]
        oauth_response = oauth.parse_authorization_response(url)

        access_token_url = "https://api.tripit.com/oauth/access_token"
        oauth = OAuth1Session(API_TOKEN,
            client_secret = API_SECRET,
            resource_owner_key = session["oauth_token"],
            resource_owner_secret = session["oauth_token_secret"],
            verifier = APP_NAME)

        oauth_tokens = oauth.fetch_access_token(access_token_url)

        oauth_sessions[oauth_token] = {
            "oauth_token":        oauth_tokens["oauth_token"],
            "oauth_token_secret": oauth_tokens["oauth_token_secret"]
        }

        self.redirect("/main")


class MainHandler(tornado.web.RequestHandler):        
    def get(self):
        if (self.get_secure_cookie("session_id") not in oauth_sessions):
            self.redirect("/")

        t = tornado.template.Template(open("main.html").read())
        self.write(t.generate(types=sorted(EXTRACTORS.keys())))


class OutputHandler(tornado.web.RequestHandler):
    def post(self):
        if (self.get_secure_cookie("session_id") not in oauth_sessions):
            self.redirect("/")

        if self.get_body_argument("csv", None):
            self.set_header("Content-Type", "text/csv")
            self.set_header("Content-Disposition", "attachment; filename=output.csv")

        session_id = self.get_secure_cookie("session_id")

        auth = OAuth1(
            API_TOKEN,
            API_SECRET,
            oauth_sessions[session_id]["oauth_token"],
            oauth_sessions[session_id]["oauth_token_secret"]
        )

        type_     = self.get_body_argument("activity-type")
        skip_cost = self.get_body_argument("skip-cost", "off") == "on"
        period    = self.get_body_argument("period")
        traveler  = self.get_body_argument("traveler")

        extractor = EXTRACTORS[type_]
        schema = extractor.schema
        if skip_cost:
            skip_ix = schema.index("ActivityCost")
            del schema[skip_ix]


        if self.get_body_argument("csv", None):
            f = StringIO()
            writer = csv.writer(f, lineterminator = '\n', encoding = 'utf-8')

            writer.writerow(schema)

            for trip in get_trips_from_api(auth, traveler, period):
                for row in extractor.extract(trip):
                    if skip_cost:
                        del row[skip_ix]
                    writer.writerow(row)

            self.write(f.getvalue())
        elif self.get_body_argument("datatables", None):
            t = tornado.template.Template(open("datatables.html").read(), autoescape=None)

            rows = []
            for trip in get_trips_from_api(auth, traveler, period):
                for row in extractor.extract(trip):
                    if skip_cost:
                        del row[skip_ix]
                    rows.append(row)

            self.write(t.generate(schema=schema, rows=rows))


class LogoutHandler(tornado.web.RequestHandler):
    def get(self):
        session_id = self.get_secure_cookie("session_id")
        if session_id:
            self.clear_cookie("session_id")
            del oauth_sessions[session_id]
        self.redirect("/")


def make_app():
    return tornado.web.Application([
        (r"/", LoginHandler),
        (r"/tripit", TripitHandler),
        (r"/oauth", OAuthHandler),
        (r"/main", MainHandler),
        (r"/output", OutputHandler),
        (r"/logout", LogoutHandler)
    ], debug=True, cookie_secret=os.urandom(20))


if __name__ == "__main__":
    PORT = 8888

    print("Server started. Please open your browser and go to http://localhost:%d" % PORT)
    app = make_app()
    app.listen(PORT)
    tornado.ioloop.IOLoop.current().start()

login.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <style>
      body {
        padding-top: 40px;
        padding-bottom: 40px;
        background-color: #eee;
      }

      .form-signin {
        max-width: 440px;
        padding: 15px;
        margin: 0 auto;
      }

      .form-signin .form-signin-heading,
      .form-signin {
        margin-bottom: 10px;
      }

      .form-signin .form-control {
        position: relative;
        -webkit-box-sizing: border-box;
        -moz-box-sizing: border-box;
        box-sizing: border-box;
        height: auto;
        padding: 10px;
        font-size: 16px;
      }

      .form-signin .form-control:focus {
        z-index: 2;
      }

    </style>
  </head>
  <body>
    <div class="container">
      <form class="form-signin">
        <h2 class="form-signin-heading">TripIt Activities to CSV</h2>
        <a class="btn btn-lg btn-primary btn-block" href="/tripit">Login with your TripIt account</a>
      </form>
    </div>
  </body>
</html>

main.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
    <script>
      window.onpageshow = function(event) {
        if (event.persisted) {
          window.location.reload();
        }
      };

      $(window).blur(function() {
        window.location.reload();
      });

      function disableSubmit(btn) {
        // Hide the button and put a disabled clone
        var parent = btn.parentNode;
        var clone = btn.cloneNode();
        btn.style = "display: none;";
        clone.disabled = true;
        parent.insertBefore(clone, btn);
      }
    </script>
  </head>
  <body>
    <div class="container py-3">
      <div class="d-flex">
        <a class="btn btn-danger ml-auto" href="/logout">Logout</a>
      </div>
      <div class="row">
        <div class="col">
          <form action="/output" method="post">
            <label class="font-weight-bold">Parameters</label>
            <div class="form-group row">
              <label for="activity-type" class="col-3 text-right col-form-label">Activity Type</label>
              <div class="col-3">
                <select class="form-control" name="activity-type" id="activity-type">
                  {% for type_ in types %}
                  <option value="{{type_}}">{{type_.capitalize()}}</option>
                  {% end %}
                </select>
              </div>
              <div class="col-3 text-right">Skip cost</div>
              <div clas="col-3">
                <div class="form-check">
                  <input type="checkbox" class="form-check-input position-static" name="skip-cost">
                </div>
              </div>
            </div>

            <div class="form-group row">
              <label class="col-3 text-right" for="traveler">Traveler</label>
              <div id="traveler" class="col-3">
                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="traveler" value="true" id="traveler-yes" checked>
                  <label class="form-check-label" for="traveler-yes">Yes</label>
                </div>

                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="traveler" value="false" id="traveler-no">
                  <label class="form-check-label" for="traveler-no">No</label>
                </div>

                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="traveler" value="all" id="traveler-both">
                  <label class="form-check-label" for="traveler-both">Both</label>
                </div>
              </div>

              <label class="col-3 text-right" for="period">Period</label>
              <div id="period" class="col-3">
                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="period" value="past" id="period-past" checked>
                  <label class="form-check-label" for="period-past">Past</label>
                </div>

                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="period" value="future" id="period-future">
                  <label class="form-check-label" for="period-future">Future</label>
                </div>

                <div class="form-check form-check">
                  <input class="form-check-input" type="radio" name="period" value="both" id="period-both">
                  <label class="form-check-label" for="period-both">Both</label>
                </div>
              </div>
            </div>
            <div class="form-group row" align="center">
              <div class="col-sm-12">
                <input type="submit" onclick="disableSubmit(this);" class="btn btn-primary" value="DataTables" name="datatables">
                <input type="submit" onclick="disableSubmit(this);" class="btn btn-success" value="Download as CSV" name="csv">
              </div>
            </div>
          </form>
        </div>
      </div>
    </div>
  </body>
</html>

datatables.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
    <link href="https://cdn.datatables.net/1.10.19/css/jquery.dataTables.min.css" rel="stylesheet">
    <script src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js"></script>
    <script>
      $(function() {
        $('#results').DataTable();
      });
    </script>
  </head>
  <body>
    <table id="results" class="display">
      <thead>
        <tr id="header-row">
          {% for column in schema %}
          <th>{{column}}</th>
          {% end %}
        </tr>
      </thead>
      <tbody>
        {% for row in rows %}
        <tr>
          {% for value in row %}
            <td>{{value}}</td>
          {% end %}
        </tr>
        {% end %}
      </tbody>
  </table>
  </body>
</html>

extraction.py

That's the same file as before.

Instructions

  • Replace the values of API_TOKEN and API_SECRET at line 17 and 18 of server.py with your own API Key and API Secret (between the double-quotes). You can also replace the value of APP_NAME, but that's not necessary apparently.
  • In a terminal, run the server with python2 server.py (or python server.py).
  • Open your browser and go to http://localhost:8888.
  • Make sure you're logged into your TripIt account in another tab.
  • In the app's tab, click on the big blue button to login with TripIt.
  • Choose your parameters in the form and click on DataTables or Download with CSV.
  • If the former was chosen, the results will show up in a DataTable.
  • If the latter was chosen, the file output.csv containing the results will be downloaded.

Edit 1: I had a massive misunderstanding concerning the usage of the API token/secret pair, but it all became clear once I took a step back. It's actually not each user that owns their own API token/secret, it's just the app itself, which in turn is owned by one user, who's meant to be the developer of that app. This simplified the code a fair bit.

Edit 2: Added DataTables and cleaned up main.html.

To answer your questions above: 1. Yes, the form once you're logged in has all the same parameters that the switches provided before. However, I have removed the cache functionality, as it must have been too confusing to use, with little to no benefit. 2. That wasn't part of the original bounty, but I can make it so the app outputs a DataTables-enabled table, and it wouldn't be very hard. I'll do it if you award me the bounty. 3. In theory, all you need is a server and something like an AWS EC2 instance would do. To make things more proper, HTTPS should be used and the session mechanism should be made more robust. The app could also be adapted to be split into separate AWS Lambda functions. Send me a personal message if you want to know more (click on my name, and then on "Contact").
CyteBode 3 months ago