Fluentd output plugin for inserting data to BigObject
Installation
Add this line to your application's Gemfile:
gem 'fluent-plugin-bigobject'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-bigobject
Usage
Configure BigObject URL and the table/column to be mapped in BigObject
#sample source to read csv file <source> type tail #path- where you placed your input data path ./input/Customer.csv # pos_file where you record file position pos_file ./log/customer.log.pos # for bigobject output plugin, use tag bigobject.${table_pattern}.${event}.${primary_key} # ${primary_key} is not needed for insert tag bigobject.cust.insert #input file format format csv # keys - columns in csv file keys id,name,language,state,company,gender,age #types - string/bool/integer/float/time/array types age1:integer </source> # Send data to BigObject using Restful API. Tables need to be created in advance in BigObject. # depending on the event in tag received, will send data to BigObject for insert/update/delete. # # Tag for each event - bigobject.${table_pattern}.${event}.${primary_key}. # ${table_pattern} : will match to the <pattern> in <table> section of bigobject output plugin # ${event} : valid event type by insert/update/delete. # ${primary_key} : the primary key for table, optional for insert event. # if primary_key is integer type in BigObject, set bo_primary_key_is_int to true # # Eg: # tag bigobject.cust.insert ==> INSERT INTO <table> VALUES ... # tag bigobject.cust.delete.id ==> DELETE FROM <table> WHERE id=... # tag bigobject.cust.update.id ==> UPDATE <table> SET ... WHERE id=... <match bigobject.**> type bigobject log_level info # specify the bigobject host/port to connect to bigobject_hostname 192.168.59.103 bigobject_port 9090 remove_tag_prefix bigobject. flush_interval 60s <table> table Customer pattern cust #optional- #bo_primary_key_is_int true #defualts to false #column_mapping id,name,language,state,company,gender,age #bo_workspace #bo_opts </table> </match>
License
The gem is available as open source under the terms of the MIT License.