hub / github.com/alibaba/DataX
github.com/alibaba/DataX @main sqlite
6,661 symbols
22,494 edges
869 files
853 documented · 13%
README
DataX
DataX 是阿里云 DataWorks数据集成 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS, databend 等各种异构数据源之间高效的数据同步功能。
DataX 商业版本
阿里云DataWorks数据集成是DataX团队在阿里云上的商业化产品，致力于提供复杂网络环境下、丰富的异构数据源之间高速稳定的数据移动能力，以及繁杂业务背景下的数据同步解决方案。目前已经支持云上近3000家客户，单日同步数据超过3万亿条。DataWorks数据集成目前支持离线50+种数据源，可以进行整库迁移、批量上云、增量同步、分库分表等各类同步解决方案。2020年更新实时同步能力，支持10+种数据源的读写任意组合。提供MySQL，Oracle等多种数据源到阿里云MaxCompute，Hologres等大数据引擎的一键全增量同步解决方案。
商业版本参见： https://www.aliyun.com/product/bigdata/ide
Features
DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。
DataX详细介绍
请参考：DataX-Introduction
Quick Start
Download DataX下载地址
请点击：Quick Start
Support Data Channels
DataX目前已经有了比较全面的插件体系，主流的RDBMS数据库、NOSQL、大数据计算系统都已经接入，目前支持数据如下图，详情请点击：DataX数据源参考指南
| 类型 | 数据源 | Reader(读) | Writer(写) | 文档 |
|---|---|---|---|---|
| RDBMS 关系型数据库 | MySQL | √ | √ | 读 、写 |
| Oracle | √ | √ | 读 、写 | |
| OceanBase | √ | √ | 读 、写 | |
| SQLServer | √ | √ | 读 、写 | |
| PostgreSQL | √ | √ | 读 、写 | |
| DRDS | √ | √ | 读 、写 | |
| Kingbase | √ | √ | 读 、写 | |
| 通用RDBMS(支持所有关系型数据库) | √ | √ | 读 、写 | |
| 阿里云数仓数据存储 | ODPS | √ | √ | 读 、写 |
| ADB | √ | 写 | ||
| ADS | √ | 写 | ||
| OSS | √ | √ | 读 、写 | |
| OCS | √ | 写 | ||
| Hologres | √ | 写 | ||
| AnalyticDB For PostgreSQL | √ | 写 | ||
| 阿里云中间件 | datahub | √ | √ | 读 、写 |
| SLS | √ | √ | 读 、写 | |
| 图数据库 | 阿里云 GDB | √ | √ | 读 、写 |
| Neo4j | √ | 写 | ||
| NoSQL数据存储 | OTS | √ | √ | 读 、写 |
| Hbase0.94 | √ | √ | 读 、写 | |
| Hbase1.1 | √ | √ | 读 、写 | |
| Phoenix4.x | √ | √ | 读 、写 | |
| Phoenix5.x | √ | √ | 读 、写 | |
| MongoDB | √ | √ | 读 、写 | |
| Cassandra | √ | √ | 读 、写 | |
| 数仓数据存储 | StarRocks | √ | √ | 读 、写 |
| ApacheDoris | √ | 写 | ||
| ClickHouse | √ | √ | 读 、写 | |
| Databend | √ | 写 | ||
| Hive | √ | √ | 读 、写 | |
| kudu | √ | 写 | ||
| selectdb | √ | 写 | ||
| 无结构化数据存储 | TxtFile | √ | √ | 读 、写 |
| FTP | √ | √ | 读 、写 | |
| HDFS | √ | √ | 读 、写 | |
| Elasticsearch | √ | 写 |
Extension points exported contracts — how you extend this code
IOtsReaderSlaveProxy (Interface)OTS Reader工作进程接口 [9 implementers]
otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderSlaveProxy.java
Record (Interface)
Created by jingxing on 14-8-24. [7 implementers]
common/src/main/java/com/alibaba/datax/common/element/Record.java
DorisCodec (Interface)
(no doc) [6 implementers]
doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisCodec.java
UnstructuredWriter (Interface)
(no doc) [6 implementers]
plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage/writer/UnstructuredWriter.java
IOtsWriterSlaveProxy (Interface)
(no doc) [6 implementers]
otswriter/src/main/java/com/alibaba/datax/plugin/writer/otswriter/IOtsWriterSlaveProxy.java
SelectdbCodec (Interface)
(no doc) [6 implementers]
selectdbwriter/src/main/java/com/alibaba/datax/plugin/writer/selectdbwriter/SelectdbCodec.java
StarRocksISerializer (Interface)
(no doc) [7 implementers]
starrockswriter/src/main/java/com/starrocks/connector/datax/plugin/writer/starrockswriter/row/StarRocksISerializer.java
IObPartCalculator (Interface)
@author cjyyz @date 2023/02/07 @since [4 implementers]
oceanbasev10writer/src/main/java/com/alibaba/datax/plugin/writer/oceanbasev10writer/part/IObPartCalculator.java
Core symbols most depended-on inside this repo
getStringcalled by 1217
common/src/main/java/com/alibaba/datax/common/util/Configuration.java
asDataXException
called by 1204
common/src/main/java/com/alibaba/datax/common/exception/DataXException.java
info
called by 749
loghubwriter/src/main/java/com/alibaba/datax/plugin/writer/loghubwriter/LogHubWriter.java
add
called by 681
gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/model/GdbGraph.java
append
called by 620
plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage/reader/binaryFileUtil/ByteUtils.java
size
called by 620
core/src/main/java/com/alibaba/datax/core/transport/channel/Channel.java
toString
called by 442
common/src/main/java/com/alibaba/datax/common/element/Record.java
put
called by 438
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/Connection4TSDB.java
Shape
Languages
Java99%
Python1%
Modules by API surface
common/src/main/java/com/alibaba/datax/common/statistics/PerfTrace.java115 symbols
hbase11xsqlwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xsqlwriter/ThinClientPTable.java72 symbols
oceanbasev10writer/src/main/java/com/alibaba/datax/plugin/writer/oceanbasev10writer/directPath/AbstractRestrictedPreparedStatement.java71 symbols
common/src/main/java/com/alibaba/datax/common/util/Configuration.java53 symbols
otswriter/src/main/java/com/alibaba/datax/plugin/writer/otswriter/model/OTSConf.java50 symbols
odpswriter/src/main/java/com/alibaba/datax/plugin/writer/odpswriter/util/OdpsUtil.java48 symbols
otsstreamreader/src/main/java/com/alibaba/datax/plugin/reader/otsstreamreader/internal/config/OTSStreamReaderConfig.java45 symbols
otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSConf.java40 symbols
otsstreamreader/tools/tabulate.py38 symbols
oceanbasev10writer/src/main/java/com/alibaba/datax/plugin/writer/oceanbasev10writer/task/ConcurrentTableWriterTask.java38 symbols
oceanbasev10writer/src/main/java/com/alibaba/datax/plugin/writer/oceanbasev10writer/directPath/AbstractRestrictedConnection.java38 symbols
osswriter/src/main/java/com/alibaba/datax/plugin/writer/osswriter/OssWriter.java37 symbols
Dependencies from manifests, versioned
ch.qos.logback:logback-classic1×
com.alibaba.cloud.analyticdb:adb4pgclient1.0.0 · 1×
com.alibaba.cloud.analyticdb:adbclient1.0.2 · 1×
com.alibaba.datax.tdenginewriter:tdenginewriter0.0.1-SNAPSHOT · 1×
com.alibaba.datax:datax-common0.0.1-SNAPSHOT · 1×
com.alibaba.datax:datax-core0.0.1-SNAPSHOT · 1×
com.alibaba.datax:datax-example-core0.0.1-SNAPSHOT · 1×
com.alibaba.datax:datax-example-streamreader0.0.1-SNAPSHOT · 1×
com.alibaba.datax:datax-transformer1×
com.alibaba.datax:hdfsreader0.0.1-SNAPSHOT · 1×
com.alibaba.datax:hdfswriter0.0.1-SNAPSHOT · 1×
com.alibaba.datax:neo4jwriter0.0.1-SNAPSHOT · 1×
Datastores touched
(mysql)Database · 1 repos
dbDatabase · 1 repos
(mongodb)Database · 1 repos
cl_testDatabase · 1 repos
dataxDatabase · 1 repos
db1Database · 1 repos
dbNameDatabase · 1 repos
dataxDatabase · 1 repos
For agents
$ claude mcp add DataX \
-- python -m otcore.mcp_server <graph>