ECL (data-centric programming language)

ECL is a declarative, data-centric programming language designed in 2000 to allow a team of programmers to process big data across a high performance computing cluster without the programmer being involved in many of the lower level, imperative decisions.[1][2]

ECL
Paradigmdeclarative, structured, data-centric
DeveloperHPCC Systems®, LexisNexis Risk Solutions
First appeared2000
Typing disciplinestatic, strong, safe
OSGNU/Linux
Websitehttp://hpccsystems.com/
Influenced by
Prolog, Pascal, SQL, Snobol4, C++, Clarion

History

ECL was initially designed and developed in 2000 by David Bayliss as an in-house productivity tool within Seisint Inc and was considered to be a ‘secret weapon’ that allowed Seisint to gain market share in its data business. Equifax had an SQL-based process for predicting who would go bankrupt in the next 30 days, but it took 26 days to run the data. The first ECL implementation solved the same problem in 6 minutes. The technology was cited as a driving force behind the acquisition of Seisint by LexisNexis and then again as a major source of synergies when LexisNexis acquired ChoicePoint Inc.[3]

Language constructs

ECL, at least in its purest form, is a declarative, data-centric language. Programs, in the strictest sense, do not exist. Rather an ECL application will specify a number of core datasets (or data values) and then the operations which are to be performed on those values.

Hello world

ECL is to have succinct solutions to problems and sensible defaults. The "Hello World" program is characteristically short:

'Hello World'

Perhaps a more flavorful example would take a list of strings, sort them into order, and then return that as a result instead.

// First declare a dataset with one column containing a list of strings
// Datasets can also be binary, CSV, XML or externally defined structures

D := DATASET([{'ECL'},{'Declarative'},{'Data'},{'Centric'},{'Programming'},{'Language'}],{STRING Value;});
SD := SORT(D,Value);
output(SD)

The statements containing a := are defined in ECL as attribute definitions. They do not denote an action; rather a definition of a term. Thus, logically, an ECL program can be read: "bottom to top"

OUTPUT(SD)

What is an SD?

 SD := SORT(D,Value);

SD is a D that has been sorted by ‘Value’

What is a D?

 D := DATASET([{'ECL'},{'Declarative'},{'Data'},{'Centric'},{'Programming'},{'Language'}],{STRING Value;});

D is a dataset with one column labeled ‘Value’ and containing the following list of data.

ECL primitives

ECL primitives that act upon datasets include SORT, ROLLUP, DEDUP, ITERATE, PROJECT, JOIN, NORMALIZE, DENORMALIZE, PARSE, CHOSEN, ENTH, TOPN, DISTRIBUTE

ECL encapsulation

Whilst ECL is terse and LexisNexis claims that 1 line of ECL is roughly equivalent to 120 lines of C++, it still has significant support for large scale programming including data encapsulation and code re-use. The constructs available include MODULE, FUNCTION, FUNCTIONMACRO, INTERFACE, MACRO, EXPORT, SHARED

Support for Parallelism in ECL

In the HPCC implementation, by default, most ECL constructs will execute in parallel across the hardware being used. Many of the primitives also have a LOCAL option to specify that the operation is to occur locally on each node.

Comparison to Map-Reduce

The Hadoop Map-Reduce paradigm consists of three phases which correlate to ECL primitives as follows.

Hadoop Name/Term ECL equivalent Comments
MAPing within the MAPper PROJECT/TRANSFORM Takes a record and converts to a different format; in the Hadoop case the conversion is into a key-value pair
SHUFFLE (Phase 1) DISTRIBUTE(,HASH(KeyValue)) The records from the mapper are distributed depending upon the KEY value
SHUFFLE (Phase 2) SORT(,LOCAL) The records arriving at a particular reducer are sorted into KEY order
REDUCE ROLLUP(,Key,LOCAL) The records for a particular KEY value are now combined
gollark: They were owned by MS for *ages*.
gollark: I do somewhat expect people to come up with alternative authentication mechanisms, but MS would also probably complain.
gollark: GPT-██ trained using people's souls as well as that stuff, actually, see potatOS privacy policy clause 4.7.
gollark: Does EndOS contain any actual code?
gollark: XTMF has more than length, though.

References
  1. A Guide to ECL, Lexis-Nexis.
  2. "Evaluating use of data flow systems for large graph analysis," by A. Yoo, and I. Kaplan. Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS, 2009
  3. "Acquisition of Seisint". Archived from the original on 2011-06-21. Retrieved 2011-03-24.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.