{"activeVersionTag":"latest","latestAvailableVersionTag":"latest","collection":{"info":{"_postman_id":"d1d74838-8040-4bea-ad72-14c7a3ec5a5a","name":"MTNA Rich Data Services","description":"Rich Data Services (RDS) is a suite of REST APIs designed by Metadata Technology North America (MTNA) to meet various needs for data engineers, managers, custodians, and consumers. RDS provides a range of services including data profiling, mapping, transformation, validation, ingestion, and dissemination. For more information about each of these APIs and how you can incorporate or consume them as part of your work flow please visit the [MTNA website](https://mtna.us/).\n\n# Overview\nThe APIs documented here provide data managers a way to expose and disseminate their data in useful, robust, and flexible ways to end users. Data managers can use RDS to not only set up connections to their data, but document it in ways that enlighten users as to the original purpose of the data and its intended use, making data discovery for the end user much more convenient. \n\n## Exposing Data\nManagers begin by building their RDS catalog, a two level hierarchy of catalogs and data products. Each catalog could be a clients data, a series of data products, or even data products about similar topics. The catalogs help managers group data products in a logical fashion. After a catalog is created, data products can be added to them, which set up the specifics of the data source that is being used. Once the data product is added to the catalog, the manager can choose to be done. The basic metadata from the data source will be used to fill out information about the variables in the data product (ID, name, data type) and the data can be queried by end users immediately. If a manager does not wish to expose their catalog or data products to anyone other than administrators, these can be set to \"private\".\n\n## Enhancing Data With Metadata\nWhile data can be queried immediately after setting up a data product, managers can continue to enhance the data product through the process of documenting the data (creating metadata). This could be updating variable names to be more meaningful, adding labels, descriptions, classifications, or one of many other properties on the variables in the data product. The addition of metadata helps bring meaning and intention to the data. This is vital for robust user interfaces, processing, and even the generated data packages that RDS performs. In addition, metadata allows users to discover the data they are looking for easily.\n\n## Rich Querying\nVery rarely will users actually want to view an entire data set, most of the time it is expected for users to subset their data based on variables or topics of interest. This brings up a couple interesting issues for users trying to access the data. If users are unfamiliar with the data source or variable names, how are they supposed to select the variables they are interested in? This may not be a big deal for small data sources, but do we really expect users to search through hundreds or thousands of variables on big data sources in order to find the variable they are interested in? To solve this problem, RDS leverages the documentation that administrators have added to provide enhanced selection capability. \n\n### Query by Variable Names / Regular Expression\nThe first way RDS allows users to select variables is through the well known and traditional way of specifying variable names. If we know the variables we want to work with this is a great way to quickly subset our data. This is done through the use of the cols parameter, which takes a comma separated list of values. The names used are all treated as regular expressions, so feel free to treat them as such. \n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/select?cols=col1,col2,col3,col[0-9]+,... `\n\n### Query by Keyword\nKey words are another way to search for variables. When selecting variables by keywords it is important to have good metadata backing the variables. A variables name, label, description, and question text will be searched for the keyword and any and all matches will be returned. To search on keywords we will have to put a $ before the keyword we would like to search on in the cols parameter.\n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/select?cols=$keyword`\n\n## Interactive Dictionaries\nWhile RDS is designed to deliver both data and metadata, the metadata can be queried by itself in order to provide clients with a wealth of knowledge about what is in the data. A dynamic data dictionary could be created to provide users with an in depth look at a data product before they even begin to query the data. \n\n### Getting the Variables\nThe logical place to start when displaying a data products contents would be the products variables (columns). This will provide end users with high level information about the variables which they can then drill down into if desired. RDS provides variable summaries as a quick way to set this up. The in addition to variable information, the summary also may contain a classification URI, which could be used to direct a user to a particular classification without the need to dig any further into the variable metadata.\n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/variables`\n\n### Getting More Information About a Variable\nThe variable summaries are a good place to start when presenting the metadata to users, but to save querying time they are not used to store all of the variable information (summary statistics and frequencies for instance). If a user selects a variable, it would make sense to get a full variable with all the information available.\n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/variable/{variableId}`\n\n### Getting a Classification\nClassifications are essentially a set of codes and their values, and are often vital to understanding the data stored in any given data product. If a variable has a classification associated with it it will have the 'classification URI' property filled out. This can be used to retrieve high level information about the classification. Beware though, due to the fact that some classifications can be extremely large (100ks of codes) no codes will be returned on the classification, they need to be queried by themselves. \n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/classification/{classificationUri}`\n\n### Building a Classification\nDue to the fact that classifications can be very large, separate calls must be made to get a subset of codes in batches. These can be used to build out a classification as a user browsers / paginates through the codelist. \n\n`http://{host}/rds/api/catalog/{catalogId}/{dataProductId}/classification/{classificationUri}/codes`\n\n\n## Customized Data Packaging\nIn many instances, data available on the web is provided in an open format (CSV or tab delimited) along with documentation typically provided in text, Word, or PDF formats. This presents a huge challenge for users as these formats are not machine actionable, meaning that the user has the burden of transcribing the metadata into the format they want. \n\nRDS solves this by providing a variety of formats that include the metadata in actionable ways. In addition to the text or Excel data formats that are so common, SQL scripts can be generated to recreate the extract (and variables look up tables) in the users preferred SQL database. For users that prefer to load their data in statistical software, SAS, SPSS, and Stata syntax can be generated to accompany the data.\n\n### Discovering Packaging Formats\nCurious as to what formats RDS provides? Just make the following call to discover what is available, the formats will be organized into sections for data, documentation, metadata, and scripts.\n\n`http://{host}/rds/api/package/options`\n\nThe returned formats should have a name, ID, and description, giving a basic description of what the formats are and their intended purpose. \n\n### Configure Packaging Options\nWhen users find a package that fits their need, they can get a configuration object using the ID as follows:\n\n`http://{host}/rds/api/package/options/{formatId}`\n\nThis will return an object containing all the options available for the format. These options can filled out and added to an array of packaging options that can be used to create a data package. \n\n### Packaging Data\nOnce all the formats are picked and their options filled out, these can be posted to the following endpoints (with the appropriate queries in the query parameters as normal) to start a packaging process.\n\n`http://{host}/rds/api/package/{catalogId}/{dataProductId}/select`\n`http://{host}/rds/api/package/{catalogId}/{dataProductId}/tabulate`\n\nThese will return a process ID, which can be used to check the process status at \n\n`http://{host}/rds/api/package/processdetails/{processId}`\n\nand to download the package upon completion at\n\n`http://{host}/rds/api/package/{processId}`\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","isPublicCollection":false,"owner":"2220438","collectionId":"d1d74838-8040-4bea-ad72-14c7a3ec5a5a","publishedId":"S1EH32U4","public":true,"publicUrl":"https://documenter-api.postman.tech/view/2220438/S1EH32U4","privateUrl":"https://go.postman.co/documentation/2220438-d1d74838-8040-4bea-ad72-14c7a3ec5a5a","customColor":{"top-bar":"FFFFFF","right-sidebar":"106FA5","highlight":"89BC44"},"documentationLayout":"classic-double-column","customisation":null,"version":"8.10.1","publishDate":"2020-05-21T19:45:52.000Z","activeVersionTag":"latest","documentationTheme":"light","metaTags":{},"logos":{}},"statusCode":200},"environments":[{"name":"RDS Documentation","id":"0463083e-7be6-422d-859b-877489f9bdae","owner":"2220438","values":[{"key":"baseUrl","value":"host","enabled":true},{"key":"catalogId","value":"catalogId","enabled":true},{"key":"dataProductId","value":"dataProductId","enabled":true},{"key":"host","value":"http://covid19.richdataservices.com","enabled":true}],"published":true}],"user":{"authenticated":false,"permissions":{"publish":false}},"run":{"button":{"js":"https://run.pstmn.io/button.js","css":"https://run.pstmn.io/button.css"}},"web":"https://www.getpostman.com/","team":{"logo":"https://res.cloudinary.com/postman/image/upload/t_team_logo_pubdoc/v1/team/768118b36f06c94b0306958b980558e6915839447e859fe16906e29d683976f0","favicon":""},"isEnvFetchError":false,"languages":"[{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"HttpClient\"},{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"RestSharp\"},{\"key\":\"curl\",\"label\":\"cURL\",\"variant\":\"cURL\"},{\"key\":\"dart\",\"label\":\"Dart\",\"variant\":\"http\"},{\"key\":\"go\",\"label\":\"Go\",\"variant\":\"Native\"},{\"key\":\"http\",\"label\":\"HTTP\",\"variant\":\"HTTP\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"OkHttp\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"Unirest\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"Fetch\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"jQuery\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"XHR\"},{\"key\":\"c\",\"label\":\"C\",\"variant\":\"libcurl\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Axios\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Native\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Request\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Unirest\"},{\"key\":\"objective-c\",\"label\":\"Objective-C\",\"variant\":\"NSURLSession\"},{\"key\":\"ocaml\",\"label\":\"OCaml\",\"variant\":\"Cohttp\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"cURL\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"Guzzle\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"HTTP_Request2\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"pecl_http\"},{\"key\":\"powershell\",\"label\":\"PowerShell\",\"variant\":\"RestMethod\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"http.client\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"Requests\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"httr\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"RCurl\"},{\"key\":\"ruby\",\"label\":\"Ruby\",\"variant\":\"Net::HTTP\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"Httpie\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"wget\"},{\"key\":\"swift\",\"label\":\"Swift\",\"variant\":\"URLSession\"}]","languageSettings":[{"key":"csharp","label":"C#","variant":"HttpClient"},{"key":"csharp","label":"C#","variant":"RestSharp"},{"key":"curl","label":"cURL","variant":"cURL"},{"key":"dart","label":"Dart","variant":"http"},{"key":"go","label":"Go","variant":"Native"},{"key":"http","label":"HTTP","variant":"HTTP"},{"key":"java","label":"Java","variant":"OkHttp"},{"key":"java","label":"Java","variant":"Unirest"},{"key":"javascript","label":"JavaScript","variant":"Fetch"},{"key":"javascript","label":"JavaScript","variant":"jQuery"},{"key":"javascript","label":"JavaScript","variant":"XHR"},{"key":"c","label":"C","variant":"libcurl"},{"key":"nodejs","label":"NodeJs","variant":"Axios"},{"key":"nodejs","label":"NodeJs","variant":"Native"},{"key":"nodejs","label":"NodeJs","variant":"Request"},{"key":"nodejs","label":"NodeJs","variant":"Unirest"},{"key":"objective-c","label":"Objective-C","variant":"NSURLSession"},{"key":"ocaml","label":"OCaml","variant":"Cohttp"},{"key":"php","label":"PHP","variant":"cURL"},{"key":"php","label":"PHP","variant":"Guzzle"},{"key":"php","label":"PHP","variant":"HTTP_Request2"},{"key":"php","label":"PHP","variant":"pecl_http"},{"key":"powershell","label":"PowerShell","variant":"RestMethod"},{"key":"python","label":"Python","variant":"http.client"},{"key":"python","label":"Python","variant":"Requests"},{"key":"r","label":"R","variant":"httr"},{"key":"r","label":"R","variant":"RCurl"},{"key":"ruby","label":"Ruby","variant":"Net::HTTP"},{"key":"shell","label":"Shell","variant":"Httpie"},{"key":"shell","label":"Shell","variant":"wget"},{"key":"swift","label":"Swift","variant":"URLSession"}],"languageOptions":[{"label":"C# - HttpClient","value":"csharp - HttpClient - C#"},{"label":"C# - RestSharp","value":"csharp - RestSharp - C#"},{"label":"cURL - cURL","value":"curl - cURL - cURL"},{"label":"Dart - http","value":"dart - http - Dart"},{"label":"Go - Native","value":"go - Native - Go"},{"label":"HTTP - HTTP","value":"http - HTTP - HTTP"},{"label":"Java - OkHttp","value":"java - OkHttp - Java"},{"label":"Java - Unirest","value":"java - Unirest - Java"},{"label":"JavaScript - Fetch","value":"javascript - Fetch - JavaScript"},{"label":"JavaScript - jQuery","value":"javascript - jQuery - JavaScript"},{"label":"JavaScript - XHR","value":"javascript - XHR - JavaScript"},{"label":"C - libcurl","value":"c - libcurl - C"},{"label":"NodeJs - Axios","value":"nodejs - Axios - NodeJs"},{"label":"NodeJs - Native","value":"nodejs - Native - NodeJs"},{"label":"NodeJs - Request","value":"nodejs - Request - NodeJs"},{"label":"NodeJs - Unirest","value":"nodejs - Unirest - NodeJs"},{"label":"Objective-C - NSURLSession","value":"objective-c - NSURLSession - Objective-C"},{"label":"OCaml - Cohttp","value":"ocaml - Cohttp - OCaml"},{"label":"PHP - cURL","value":"php - cURL - PHP"},{"label":"PHP - Guzzle","value":"php - Guzzle - PHP"},{"label":"PHP - HTTP_Request2","value":"php - HTTP_Request2 - PHP"},{"label":"PHP - pecl_http","value":"php - pecl_http - PHP"},{"label":"PowerShell - RestMethod","value":"powershell - RestMethod - PowerShell"},{"label":"Python - http.client","value":"python - http.client - Python"},{"label":"Python - Requests","value":"python - Requests - Python"},{"label":"R - httr","value":"r - httr - R"},{"label":"R - RCurl","value":"r - RCurl - R"},{"label":"Ruby - Net::HTTP","value":"ruby - Net::HTTP - Ruby"},{"label":"Shell - Httpie","value":"shell - Httpie - Shell"},{"label":"Shell - wget","value":"shell - wget - Shell"},{"label":"Swift - URLSession","value":"swift - URLSession - Swift"}],"layoutOptions":[{"value":"classic-single-column","label":"Single Column"},{"value":"classic-double-column","label":"Double Column"}],"versionOptions":[],"environmentOptions":[{"value":"0","label":"No Environment"},{"label":"RDS Documentation","value":"2220438-0463083e-7be6-422d-859b-877489f9bdae"}],"canonicalUrl":"https://documenter.gw.postman.com/view/metadata/S1EH32U4"}