The Data Lab menu
  • Topicsarrow_drop_down
  • Projects
  • Aboutarrow_drop_down
  • Get Connected
  • Machine Learning for Developers
  • Notebooks for Developers
  • Offline First
  • Partners + Data
  • Serverless + Data
  • Working with JSON
  • Team
  • Get Involved
  • Code of Conduct
  • Event Support
  • Projects
  • Topicsarrow_drop_down
  • Aboutarrow_drop_down
  • Get Connected
  • Machine Learning for Developers
  • Notebooks for Developers
  • Offline First
  • Partners + Data
  • Serverless + Data
  • Working with JSON
  • Team
  • Get Involved
  • Code of Conduct
  • Event Support
Working with JSON / Collection

Loading JSON

Moving data into data services should be easy. We can help.

Glynn Bird
Making it easier to work with JSON.
More by Glynn Bird

On the IBM Cloud, it should be easy to move data in, share it between services, and get data out. These projects represent our ongoing work to make data import and export easier for users.

Importing structured data into document data stores

Structured data living in spreadsheets, relational databases, or as flat CSV/TSV files can be easily imported into IBM Cloudant, MongoDB or ElasticSearch using our nosqlimport tool.

Cloudant and CouchDB have their own tool, the popular couchimport utility which allows structured data to be bulk imported into the JSON document stores, with an optional transformation along the way. The couchimport utility is also packaged as an installable Electron app to make data transfer as simple as a “drag and drop.”

Moving data from other clouds

Developers are keen to avoid being locked in to a cloud platform so we have built tools to extract data from other cloud databases, which can easily be paired with couchimport or nosqlimport to move the data elsewhere.

Backing up Cloudant

Cloudant is an operational database, safely storing vital data for our customers in distributed clusters. Customer backup needs vary widely, but can be scripted easily with the couchbackup utility which can turn databases into text files, to allow rotating backups to be archived on cheaper object storage.

Transporter

The nice folks at Compose have written the Transporter utility that makes data movement between files and their MongoDB, ElasticSearch, RethinkDB, PostgreSQL, and RabbitMQ products.

↓ View projects in this collection

Projects

  • Elasticsearch, Bulk Uploading and the High-Level Java REST Client - Part 2
    GitHub | Compose Articles

    How to use the BulkProcessor to bulk up all your Elasticsearch updates.

    • Compose
    • Elasticsearch
    • Java
  • Elasticsearch, Bulk Uploading and the High-Level Java REST Client - Part 1
    GitHub | Compose Articles

    A practical look at the relatively new High-Level Java REST Client for Elasticsearch uploads.

    • Compose
    • Elasticsearch
    • Java
  • Cloudant and CouchDB Replication with couchreplicate
    GitHub | Medium

    A new command-line tool for managing and monitoring multiple replication jobs.

    • Cloudant
    • CouchDB
    • JavaScript
  • Tuning it up to 11 - Transporter and the Cloudant Adaptor
    GitHub | Compose Articles

    Revisiting the simple Cloudant adaptor to make it a bit smarter.

    • Compose
    • Transporter
    • Cloudant
    • Go
  • Keeping Local Documents Local in a CouchDB-Style Replication Job
    Medium

    For when you replicate data, but need to leave a bit of config behind.

    • Cloudant
    • CouchDB
    • PouchDB
  • Building the Soul of a New Transporter Adaptor
    Compose Articles

    Taking the first steps to creating our own Transporter adapter for Cloudant.

    • Compose
    • Transporter
    • Go
  • Building Your Own Data Transport with Compose Transporter
    Compose Articles

    A stodge-free open source ETL tool you can extend in Go. Here's how.

    • Compose
    • Transporter
    • Go
  • Using Your Own Domain Name With Cloudant
    GitHub | Medium

    Secure your custom domain using serverless functions on the IBM Cloud.

    • Cloudant
    • OpenWhisk
    • IBM Cloud Functions
    • Serverless
  • Building a Serverless Data Pipeline
    GitHub | Medium | YouTube

    Data transformation is a great use case for serverless technology.

    • OpenWhisk
    • IBM Cloud Functions
    • Serverless
    • Nodejs
    • Cloudant
  • Search in Reverse: Real-Time Updates for Data as It Enters Your System
    Compose Articles

    Discover space on twitter using Elasticsearch's percolate query.

    • Compose
    • Elasticsearch
    • MongoDB
    • Nodejs
  • Couchimport: Import flat data into CouchDB efficiently.
    npm | GitHub | Medium

    Latest update adds one-to-many transformations.

    • CouchDB
    • Cloudant
  • Visualizing Data with Jupyter Notebooks, PixieDust, and Compose MongoDB
    Compose Articles

    Connecting to Compose MongoDB and creating rich presentations for your data inside a Jupyter notebook is made easier with PixieDust.

    • Jupyter Notebook
    • Compose
    • MongoDB
    • PixieDust
    • Python
    • Nodejs
  • The Transporter 0.4 series has begun
    GitHub | Compose Articles

    Compose's Transporter is a powerful way of transporting data between different databases. With each iteration, we look to improve it.

    • Transporter
    • JavaScript
  • Custom Cloudant Replication
    GitHub | Medium

    Emulating filtered replication using the changes feed and OpenWhisk functions.

    • Cloudant
    • JavaScript
    • OpenWhisk
    • IBM Cloud Functions
    • Serverless
  • Diff Your Couch-Style Databases with couchdiff
    GitHub | Medium | npm

    Command-Line tool for comparing two Apache CouchDB™ or Cloudant databases.

    • CouchDB
    • Cloudant
  • Moving data from DynamoDB to Cloudant or CouchDB
    GitHub | Medium | npm

    Introducing the dynamodbexport npm module.

    • Cloudant
    • CouchDB
    • Nodejs
  • Installing Web Apps with Electron
    GitHub | Medium

    A simple CouchDB data importer app that also shows how to bundle a web app for native distribution with Electron.

    • CouchDB
    • Electron
    • Nodejs
  • Seven Databases in Seven Days
    GitHub | developerWorks

    A quick tour of (some of) our database services.

    • Cloudant
    • Compose
    • Etcd
    • Tinkerpop
    • MongoDB
    • PostgreSQL
    • Redis
    • RethinkDB
  • CouchDB Document Copy and Transform Service
    GitHub | developerWorks

    Add some T to your ETL for CouchDB.

    • Cloudant
    • CouchDB
    • Nodejs
  • Importing JSON Documents with nosqlimport
    GitHub | developerWorks | npm

    Moving data doesn't have to stink.

    • Nodejs
    • CouchDB
    • Cloudant
    • Elasticsearch
  • Simple CouchDB and Cloudant Backup
    GitHub | developerWorks | npm

    Back up your data, without breaking your back.

    • Nodejs
    • CouchDB
    • Cloudant
© 2017 IBM Watson Data Lab