IT博客汇
  • 首页
  • 精华
  • 技术
  • 设计
  • 资讯
  • 扯淡
  • 权利声明
  • 登录 注册

    perl dbi table copy oracle to mysql

    lazydba发表于 2010-05-21 21:26:33
    love 0
    并行是一个加快导数据速度的好方法,可以充分利用机器的处理能力。
    1. select from ora_table可以用extents方式切割,一个进程处理一部分extent,多个进程可以同时运行。

    2. 处理数据的进程可以充分利用多个cpu,字符串转义的处理和拼接mysql的批量insert语句需要不少cpu资源。
    ?? perl可以用fork生成多个进程, $degree是并行度:

    ??? for (my $t = 0; $t < $degree; $t ++) {
    ??????? if ($pid = fork()) {
    ??????????? #parent,
    ??????????? $slaves[$t] = $pid;
    ??????? } elsif (defined $pid) {
    ????????? ? #child 数据库连接要在每个子进程生成.
    ????????
    ??????? ??? my $from_db = &get;_oracle_connection("tns", "user", "passwd");
    ??????????? my $to_db = &get;_mysql_connection("ip", 3306, "user", "passwd");
    ???????????
    ??????????? my $start = int($t / $degree * $num_of_extents) + 1;
    ??????????? my $end = int(($t+1) / $degree * $num_of_extents) ;
    ???????????
    ??????????? my $extents = &get;_extents_by_range($to_db, $start, $end);

    ??????????? for my $ext (@$extents) {
    ??????????????? my $rowid_min = $ext->[1];
    ??????????????? my $rowid_max = $ext->[2];

    ??????????????? &do;_copy_by_extent($from_db, $to_db,
    ??????????????????? $from_owner, $table_name,
    ??????????????????? $to_owner, $to_table,
    ??????????????????? $str_sql_select, $str_sql_insert,
    ??????????????????? $rowid_min, $rowid_max);
    ??????????? }
    ??????????? $from_db->disconnect();
    ??????????? $to_db->disconnect();
    ??????????? exit;
    ??????? } else {
    ??????????? print "fork failed: $!\n";
    ??????? }

    3. 写入mysql时可以多个进程并行写入。


    附带一个取extents的语句,可以处理分区表和非分区表

    select rownum rn,
    ??? dbms_rowid.rowid_create(1, data_object_id, relative_fno, block_id, 0) s,???????
    ??? dbms_rowid.rowid_create(1, data_object_id, relative_fno, block_id + blocks - 1, 10000) e?
    from (
    ??? select extent_id,? relative_fno, block_id, blocks, nvl(partition_name, 'default') partition_name
    ??? from dba_extents where owner = upper(?)and segment_name = upper(?)) a,
    ??? (select nvl(subobject_name, 'default') subobject_name, data_object_id
    ??????? from dba_objects where? owner=upper(?)
    ??????? and? object_name = upper(?)) b
    where a.partition_name = b.subobject_name.

    查询的时候加上rowid hint

    select /*+ rowid(t) */ * from ora_tab t where rowid >= chartorowid(?) and rowid <= chartorowid(?).




沪ICP备19023445号-2号
友情链接