#876364 dovecot-sieve: Just discovered imap_sieve/sieve_imapsieve is not set up to work with virtual mailboxes.

Package:
dovecot-sieve
Source:
dovecot
Description:
secure POP3/IMAP server - Sieve filters support
Submitter:
Thurgood Angelou
Date:
2017-10-21 23:51:09 UTC
Severity:
normal
#876364#5
Date:
2017-09-21 10:56:14 UTC
From:
To:
Dear Maintainer,

I've just discovered a bug where the sieve plugin (especially IMAP) will not work with a virtual mailbox. I recently found this out when testing a virtual mailbox setup and I use imap_sieve for SPAM filtering so to lose that would be painful.

This is what is reported in dovecot's debug log...

Sep 21 10:53:01 imap(personal@othersider92.com): Panic: file mail-index-map.c: line 549 (mail_index_map_lookup_seq_range): assertion failed: (first_uid > 0)
Sep 21 10:53:01 imap(personal@othersider92.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x9f0a2) [0x7f4226fd50a2] -> /usr/lib/dovecot/libdovecot.so.0(+0x9f19a) [0x7f4226fd519a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4226f65cf8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_map_lookup_seq_range+0x120) [0x7f422733dae0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xe8d1d) [0x7f422734cd1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_lookup_seq+0xf) [0x7f4227350e9f] -> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x82aa) [0x7f422678a2aa] -> /usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so(+0x708c) [0x7f422657d08c] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x52) [0x7f42272adca2] -> dovecot/imap(+0x10084) [0x556502adb084] -> dovecot/imap(command_exec+0x5c) [0x556502ae7d9c] -> dovecot/imap(+0x1b322) [0x556502ae6322] -> dovecot/imap(+0x1b3bc) [0x556502ae63bc] -> dovecot/imap(client_handle_input
 +0x18d) [0x556502ae677d] -> dovecot/imap(client_input+0xac) [0x556502ae6ccc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4226fea742] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) [0x7f4226febd8e] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) [0x7f4226fea7d6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4226fea988] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4226f70353] -> dovecot/imap(main+0x329) [0x556502ad9139] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f4226bb92e1] -> dovecot/imap(_start+0x2a) [0x556502ad92ca]
Sep 21 10:53:01 imap(personal@othersider92.com): Fatal: master: service(imap): child 14102 killed with signal 6 (core dumps disabled)

I switched back to a regular setup for now but I want to switch to a virtual mailbox in the future and I can't do that with imap_sieve broken like this.

Is it fixable? I really want to use IMAP sieve in absense of dovecot_antispam which does not work anymore on this debian version yet I can't do it.
#876364#10
Date:
2017-09-21 11:47:25 UTC
From:
To:
[Forwarding this to the Dovecot mailing list, just in case someone can help]

Hi,

Thanks for the report! See my comments inline.

After mixing in the debug symbols, this backtrace becomes:

/usr/lib/dovecot/libdovecot.so.0(+0x9f0a2) [0x7f4226fd50a2]
    default_fatal_finish at ./src/lib/failures.c:195
/usr/lib/dovecot/libdovecot.so.0(+0x9f19a) [0x7f4226fd519a]
    ?? at ./src/lib/failures.c:670
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4226f65cf8]
    i_fatal at ./src/lib/failures.c:280
/usr/lib/dovecot/libdovecot-storage.so.0(mail_index_map_lookup_seq_range+0x120) [0x7f422733dae0]
    mail_index_map_lookup_seq_range at ./src/lib-index/mail-index-map.c:550
/usr/lib/dovecot/libdovecot-storage.so.0(+0xe8d1d) [0x7f422734cd1d]
    tview_lookup_seq_range at ./src/lib-index/mail-index-transaction-view.c:178
/usr/lib/dovecot/libdovecot-storage.so.0(mail_index_lookup_seq+0xf) [0x7f4227350e9f]
    mail_index_lookup_seq at ./src/lib-index/mail-index-view.c:522
/usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x82aa) [0x7f422678a2aa]
    virtual_mail_set_uid at ./src/plugins/virtual/virtual-mail.c:213
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so(+0x708c) [0x7f422657d08c]
    imap_sieve_mailbox_transaction_run at ./pigeonhole/src/plugins/imapsieve/imap-sieve-storage.c:752 (inlined by: imap_sieve_mailbox_transaction_commit at ./pigeonhole/src/plugins/imapsieve/imap-sieve-storage.c:807)
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x52) [0x7f42272adca2]
    mailbox_transaction_commit_get_changes at ./src/lib-storage/mail-storage.c:2083
dovecot/imap(+0x10084) [0x556502adb084]
    cmd_copy_full at ./src/imap/cmd-copy.c:146
dovecot/imap(command_exec+0x5c) [0x556502ae7d9c]
    command_exec at ./src/imap/imap-commands.c:200
dovecot/imap(+0x1b322) [0x556502ae6322]
    client_command_input at ./src/imap/imap-client.c:1088
dovecot/imap(+0x1b3bc) [0x556502ae63bc]
    client_command_input at ./src/imap/imap-client.c:1150
dovecot/imap(client_handle_input+0x18d) [0x556502ae677d]
    client_handle_input at ./src/imap/imap-client.c:1203
dovecot/imap(client_input+0xac) [0x556502ae6ccc]
    client_input at ./src/imap/imap-client.c:1249
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4226fea742]
    io_loop_call_io at ./src/lib/ioloop.c:600
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) [0x7f4226febd8e]
    io_loop_handler_run_internal at ./src/lib/ioloop-epoll.c:223
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) [0x7f4226fea7d6]
    io_loop_handler_run at ./src/lib/ioloop.c:648
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4226fea988]
    io_loop_run at ./src/lib/ioloop.c:622
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4226f70353]
    master_service_run at ./src/lib-master/master-service.c:667
dovecot/imap(main+0x329) [0x556502ad9139]
    main at ./src/imap/main.c:491
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f4226bb92e1]
    __libc_start_main at ../csu/libc-start.c:325
dovecot/imap(_start+0x2a) [0x556502ad92ca]
    start at ??:0

Could you also include the output of doveconf -n?

Regards,
Apollon
#876364#19
Date:
2017-09-22 00:09:32 UTC
From:
To:
Hi... The output of my doveconf -n is as folllows.

# 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.20 (7cd71ba)
doveconf: Warning: /etc/dovecot/dovecot.conf line 105: Global setting
mail_plugins won't change the setting inside an earlier filter at
/etc/dovecot/conf.d/15-lda.conf line 47 (if this is intentional, avoid this
warning by moving the global setting before /etc/dovecot/conf.d/15-lda.conf
line 47)
doveconf: Warning: /etc/dovecot/dovecot.conf line 105: Global setting
mail_plugins won't change the setting inside an earlier filter at
/etc/dovecot/conf.d/15-lda.conf line 47 (if this is intentional, avoid this
warning by moving the global setting before /etc/dovecot/conf.d/15-lda.conf
line 47)
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.2 ext4
auth_default_realm = othersider92.com
auth_mechanisms = plain login
first_valid_uid = 100
log_path = /var/log/dovecot.log
mail_location = maildir:/var/mail/%d/%n
mail_plugins = " virtual"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    auto = create
    special_use = \Drafts
  }
  mailbox Junk {
    auto = create
    autoexpunge = 1 weeks
    special_use = \Junk
  }
  mailbox Promotions {
    auto = subscribe
  }
  mailbox Sent {
    auto = create
    special_use = \Sent
  }
  mailbox Trash {
    auto = create
    autoexpunge = 30 days
    special_use = \Trash
  }
  prefix =
  separator = /
  type = private
}
namespace virtual {
  disabled = yes
  hidden = yes
  list = no
  location = virtual:/var/mail/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  imapsieve_mailbox1_before = file:/etc/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = file:/etc/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  imapsieve_mailbox3_before = file:/etc/sieve/report-spam-promo.sieve
  imapsieve_mailbox3_causes = COPY
  imapsieve_mailbox3_name = Promotions
  imapsieve_mailbox4_before = file:/etc/sieve/report-ham-promo.sieve
  imapsieve_mailbox4_causes = COPY
  imapsieve_mailbox4_from = Promotions
  imapsieve_mailbox4_name = *
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_after = /etc/sieve/conf.d/after
  sieve_before = /etc/sieve/conf.d/before
  sieve_global_extensions = +editheader +vnd.dovecot.pipe
  sieve_pipe_bin_dir = /usr/local/bin
  sieve_plugins = sieve_extprograms sieve_imapsieve
}
protocols = imap lmtp
service auth-worker {
  user = vmail
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
  }
  user = dovecot
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
ssl = required
ssl_cert = </etc/ssl/mail/ssl.cert
ssl_client_ca_dir = /etc/ssl/certs
ssl_key =  # hidden, use -P to show it
userdb {
  args = uid=vmail gid=vmail home=/var/mail/%d/%n
  driver = static
}
protocol lmtp {
  info_log_path = /var/log/dovecot-lmtp.log
  mail_plugins = " virtual"
}
protocol lda {
  mail_plugins = " sieve virtual"
}
protocol imap {
  mail_plugins = " virtual imap_sieve"
}

I hope this helps.

On Thu, Sep 21, 2017 at 4:47 AM, Apollon Oikonomopoulos <apoikos@debian.org> wrote:
#876364#24
Date:
2017-09-22 00:13:10 UTC
From:
To:
Hi... The output of my doveconf -n is as folllows.

# 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.20 (7cd71ba)
doveconf: Warning: /etc/dovecot/dovecot.conf line 105: Global setting
mail_plugins won't change the setting inside an earlier filter at
/etc/dovecot/conf.d/15-lda.conf line 47 (if this is intentional, avoid this
warning by moving the global setting before /etc/dovecot/conf.d/15-lda.conf
line 47)
doveconf: Warning: /etc/dovecot/dovecot.conf line 105: Global setting
mail_plugins won't change the setting inside an earlier filter at
/etc/dovecot/conf.d/15-lda.conf line 47 (if this is intentional, avoid this
warning by moving the global setting before /etc/dovecot/conf.d/15-lda.conf
line 47)
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.2 ext4
auth_default_realm = othersider92.com
auth_mechanisms = plain login
first_valid_uid = 100
log_path = /var/log/dovecot.log
mail_location = maildir:/var/mail/%d/%n
mail_plugins = " virtual"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    auto = create
    special_use = \Drafts
  }
  mailbox Junk {
    auto = create
    autoexpunge = 1 weeks
    special_use = \Junk
  }
  mailbox Promotions {
    auto = subscribe
  }
  mailbox Sent {
    auto = create
    special_use = \Sent
  }
  mailbox Trash {
    auto = create
    autoexpunge = 30 days
    special_use = \Trash
  }
  prefix =
  separator = /
  type = private
}
namespace virtual {
  disabled = yes
  hidden = yes
  list = no
  location = virtual:/var/mail/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  imapsieve_mailbox1_before = file:/etc/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = file:/etc/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  imapsieve_mailbox3_before = file:/etc/sieve/report-spam-promo.sieve
  imapsieve_mailbox3_causes = COPY
  imapsieve_mailbox3_name = Promotions
  imapsieve_mailbox4_before = file:/etc/sieve/report-ham-promo.sieve
  imapsieve_mailbox4_causes = COPY
  imapsieve_mailbox4_from = Promotions
  imapsieve_mailbox4_name = *
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_after = /etc/sieve/conf.d/after
  sieve_before = /etc/sieve/conf.d/before
  sieve_global_extensions = +editheader +vnd.dovecot.pipe
  sieve_pipe_bin_dir = /usr/local/bin
  sieve_plugins = sieve_extprograms sieve_imapsieve
}
protocols = imap lmtp
service auth-worker {
  user = vmail
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
  }
  user = dovecot
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
ssl = required
ssl_cert = </etc/ssl/mail/ssl.cert
ssl_client_ca_dir = /etc/ssl/certs
ssl_key =  # hidden, use -P to show it
userdb {
  args = uid=vmail gid=vmail home=/var/mail/%d/%n
  driver = static
}
protocol lmtp {
  info_log_path = /var/log/dovecot-lmtp.log
  mail_plugins = " virtual"
}
protocol lda {
  mail_plugins = " sieve virtual"
}
protocol imap {
  mail_plugins = " virtual imap_sieve"
}

I hope this helps.

On Thu, Sep 21, 2017 at 4:47 AM, Apollon Oikonomopoulos <apoikos@debian.org> wrote:
#876364#29
Date:
2017-10-21 23:47:48 UTC
From:
To:
Hey.

So I just tested this with a blank (0KB) sieve script. Apparently the
plugin crashes whenever it executes the BEFORE/AFTER function and when it
checks to see if the uid's are still the same.

Copying without the plugin (tested with telnet) works but copying with the
plugin causes the plugin to crash due to the BEFORE/AFTER function being
broken for virtual folders.

Just want to know what's the ETA for a fix if at possible. I don't know how
the thing works so I'm wondering if it's at all possible to fix this simple
thing...

On Thu, Sep 21, 2017 at 5:13 PM, Thurgood Angelou <thurgoodangelou@gmail.com